Johannes Kepler University, Altenbergerstrasse 69, A-4040 Linz
knasmueller@ssw.uni-linz.ac.at
Introduction
What is Persistence
Persistence in the Oberon System
Working with
An Example
Commands of OberonD
User interface
Restrictions
anonym types
procedure variables
Implementation
Overview
General Introduction: When? Where? How?
Finalization
in Mark & Sweep
Generation of automatic Mapper
Trap
Optimiziation
Persistent Memory Management
Persistent Roots
PersTypes
Make a transient object persistent
Persistent Garbage Collection Stop & Copy
Programming Interfaces
necessary changes in the Oberon System
platform dependencies => Windows
Conclusion and Future Work
Schema evolution a.s.o.
different heaps
Appendix
Definition ISFiles.Mod
Heap format in EBNF - grammar
Error-Codes
How to get Oberon-D
References
Abstract
Oberon [ReWi92] and Oberon-2 [M
Wi91] are general purpose programming languages in the tradition of Pascal and Modula-2. But Oberon [WiGu89] is also a modular, single-threaded operating system aiming expecially at single-user operation of workstations. It is used in daily work as well as in programming courses. As our experience with Oberon shows, one missing point is the existence of database functionalities, like persistence or recovery.
This report describes the project Oberon-D, which aim is the including of database functionalities in the Oberon system. The first step is the including of persistence, the object's property to outlive the programs, that create it. The using of persistence is shown as well as the implementation.
Contents
1 Introduction
The including of database functionalities in the Oberon system is an ongoing project, named Oberon-D. In a first step Oberon-D should manage the introduction of persistence. Before beginning our work we defined different aims for this project:
As simple as possible.
No changes in the Oberon langugage.
No (less) changes in the Oberon system.
No (less) platform dependencies. There should be only one short module which has to be changed when porting Oberon-D from one platform to another (e.g. from PowerMac-Oberon to Windows-Oberon).
Minimal use of memory.
No interference with normal work. No additional delays.
Oberon_D should work like other Oberon applications. That means especially that working with persistent data should not differ from working with transient data. For example there should also be a garbage collector [.???.] for persistent data.
Persistence is an attribute describing an object`s lifetime. In a language with persistence, data objects survive between program runs. Persistence is the most important property of an object oriented database programming system. The opposite is transient, which means that transient data last only for the invocation of a program.
There are certain principles that should govern system design [AtBu87, p. 109]:
Persistence should be orthogonal, i.e. a property of arbitrary values and not limited to certain types.
All values should have the same rights to persistence.
While a value persists, so should its description (type).
Furthermore, the programmer should be able to manipulate the persistent objects with normal expression syntax, i.e. physical I/O should be transparent to the programmer.
All these mentioned points, concerning Oberon and persistence were considered by the development of Oberon-D.
2 Persistence in the Oberon System
It was clear that manipulating of persistent objects should work like manipulating of transient objects. Therefore this point must not be explained here. But there are still some interesting points: How to make an object persistent and how to address this persistent data later.
2.1 Working with persistence
By following our aim not to change the Oberon language, it was not possible to introduce a new statement to allocate persistent data, like e.g. the operator pnew in ODE [AgGe89]. Therefore we decided to take another persistent mechanism, that of reachability by a persistent root. Each allocated object can become a persistent root, by using the function Persistent.SetRoot (obj, n), where obj is an existing oberon object and n is a user-defined unique key of type String (= ARRAY OF CHAR). All objects referenced by a persistent root will be persistent objects, if there is not anything else defined (see later).
All Oberon applications can now address a root with the name n (at any time) by using the function Persistent.GetRoot (n, obj).
The following source code shows how to make a string (identified by the root myroot) persistent.
PROCEDURE MakeStringPersistent;
VAR s: POINTER TO ARRAY OF CHAR;
BEGIN
NEW (s, 32); s := "Oberon-D";
Persistent.SetRoot (s, "myroot")
END MakeStringPersistent;
To address this string after and print it to the output viewer, the following source code can be used:
PROCEDURE AddressAndPrintString;
VAR s: POINTER TO ARRAY OF CHAR;
BEGIN
Persistent.GetRoot ("myroot", s);
Out.String (s)
END AddressAndPrintString;
Making an object persistent means that the object survives the termination of the program. The only solution to make this possible is to map the object to an external representation (and back). Normally the object's properties can be written to a file in an automatic way. Oberon-D knows the structure of each object (see section
) and can therefore decide how to map the object. Unfortunately there are some problems in the case of automatic persistency (for details see [Tem94, p. 115ff]):
Closure Control (e.g Fonts)
Implicit Dependencies (e.g. File objects)
Partially used Arrays (e.g. character arrays which have a zero terminated string as contents)
Because of this reasons Oberon-D gives a user the possibility to implement a mapper for each type. This mapper must be from the structure
Mapper = PROCEDURE (o: SYSTEM.PTR);
and has to use the read and write procedures offered by the module Persistent (see section
). To order Oberon to use the self-implemented mappers one has to register them with the procedure Persistent.RegisterType (t, read, write). After such an registering operation the mappers read and write are taken instead of the automatic mappers. Be careful, it the specified type t is at record extension level n, the registered mappers are responsible for handling only the fields introduced at level n.
Another interesting point is the deleting of persistent data. In Oberon transient data is deleted by a garbage collector, which means that there is no way to explicitly dispose an allocated block. The garbage collector finds the blocks that are not used any more and makes them available for allocation again. A garbage collector frees a programmer from the non-trivial task of deallocating data structures correctly and thus helps to avoid errors.
In Oberon-D there also exists a garbage collector for persistent data. All objects which are not accessible by a persistent root are garbage and will be removed by the next run of the garbage collector, started by calling the command OberonD.Collect. You can remove a persistent root with name n by the function Persistent.RemoveRoot (n). After that the object referenced by the root n is still existing, but is only transient and not even more persistent.
2.2 An Example
The following source code shows a short example of using Oberon-D. This module offers a simple list of elements with the properties name and number and the commands Init (to allocate the list), Insert (to insert a list of elements in the list) and Print (to prints the list). Each list has a header info containing a property font, which determines the printing font. The objects of type Elem are mapped by automatic mappers, the objects of type List are mapped by the registered mappers WriteList and ReadList. These mappers must be registered by the command RegisterMappers.
MODULE Example;
IMPORT S := SYSTEM, Fonts, In, Modules, Oberon, OberonD, Out, Persistent, Texts, Types;
TYPE
List = POINTER TO ListDesc;
Elem = POINTER TO ElemDesc;
ListDesc = RECORD
first: Elem;
font: Fonts.Font;
END;
ElemDesc = RECORD
number: LONGINT;
name: ARRAY 32 OF CHAR;
next: Elem;
END;
PROCEDURE WriteList (o: S.PTR); (* writes the list o to the persistent heap *)
VAR l: List; str: ARRAY 32 OF CHAR;
BEGIN
l := S.VAL (List, o);
Persistent.WriteObj (l.first);
IF l.font # NIL THEN COPY (l.font.name, str) ELSE str := "" END;
Persistent.WriteString (str)
END WriteList;
PROCEDURE ReadList (o: S.PTR); (* reads the list o from the persistent heap *)
Module OberonD offers commands to work with roots and to check the state of the database. All tools offered by this module can also be used through a programm by calling the corresponding procedures of the module Persistent (see section
Init activates the database. This command or the corresponding procedure Persistent.Install must be called by starting the work with Oberon-D.
AllocateRoot (name t | ^) allocates a new root of type t with key name.
RemoveRoot (name | ^) removes the persistent root with key name. After that the object referenced by the root name is still existing, but is only transient and not even more persistent.
Collect starts the garbage collector of the persistent heap. This function is only available, if the database is off-line. That means garbage collection could not be started as long there are existing transient references to persistent data.
Watch shows the actual size of the persistent heap and the actual number of persistent objects.
ShowRoots shows the list of existing persistent roots.
About writes the version number of Oberon-D in the log viewer.
2.4 User Interface
The relevant procedures for a user of Oberon-D are offered by the module Persistent. This module is the programming interface to the world of Oberon-D.
PROCEDURE SetRoot (o: SYSTEM.PTR; root: ARRAY OF CHAR);
PROCEDURE Write (x: SYSTEM.BYTE);
PROCEDURE WriteBool (b: BOOLEAN);
PROCEDURE WriteInt (x: INTEGER);
PROCEDURE WriteLInt (x: LONGINT);
PROCEDURE WriteLReal (x: LONGREAL);
PROCEDURE WriteObj (o: SYSTEM.PTR);
PROCEDURE WriteProc (proc: SYSTEM.PTR);
PROCEDURE WriteReal (x: REAL);
PROCEDURE WriteSet (s: SET);
PROCEDURE WriteString (x: ARRAY OF CHAR);
END Persistent.
State
res indicates the success of an operation. If res is ok after an operation, the operation was successful and its result is valid. Otherwise an error occured. The possible error codes can be seen in appendix
Initialization routine
Install activates the database. This procedure must be called by starting the work with Oberon-D.
Reading
Read (x) reads the next byte x from the persistent heap.
ReadInt (i) reads an integer number i from the persistent heap.
ReadLInt (i) reads a long integer number i from the persistent heap.
ReadReal (x) reads a real number x from the persistent heap.
ReadLReal (x) reads a long real number x from the persistent heap.
ReadString (s) reads a sequence of character (including the terminating 0X) from the persistent heap and returns it in s. The actual paramter corresponding to s must be long enough to hold the character sequence plus the terminating 0X.
ReadSet (s) reads a set s from the persistent heap.
ReadBool (b) reads a boolean value b from the persistent heap.
ReadObj (o) reads a persistent reference to the object o from the persistent heap.
ReadProc (proc) reads the procedure proc from the persistent heap. When using ReadProc the procedure variable must be converted to the type SYSTEM.PTR with the function SYSTEM.VAL, because it is not possible to assign procedure variables to SYSTEM.PTR (see also [Tem94, p. 136]).
Writing
Write (x) writes the byte x to the persistent heap.
WriteInt (i) writes the integer number i to the persistent heap.
WriteLInt (i) writes the long integer number i to the persistent heap.
WriteReal (x) writes the real number x to the persistent heap.
WriteLReal (x) writes the long real number x to the persistent heap.
WriteString (s) writes the sequence of character s (including the terminating 0X) to the persistent heap.
WriteSet (s) writes the set s to the persistent heap.
WriteBool (b) writes the boolean value b to the persistent heap.
WriteObj (o) writes the persistent reference to the object o to the persistent heap. If o is not a persistent object, it will be marked as one.
WriteProc (proc) writes the procedure proc to the persistent heap. For the type of parameter proc see ReadProc.
Root management
GetRoot (r, o) retrieves the persistent root o identified by the key r.
SetRoot (o, r) marks the object o as a persistent root accesable through the key r.
RemoveRoot (r) removes the root identified by the key r from the root list. After that the object referenced by the root r is still existing, but is only transient and not even more persistent.
Miscellaneous
RegisterType (t, read, write) registers the read and write mappers for a transient type t.
The mappers are based on the run time data structures [M
Wi ...???] offered by the Oberon system. Oberon stores some information about a type in so-called type descriptors. This type descriptors contains for example information about the location of pointers or the type name. These two informations are in most cases enough to generate automatic mappers or to allow a programmer to install registered mappers for one type, but there are two cases in which the type desribtor contains not enough information.
There is no possibility to identify an anonym type [????] (e.g. VAR a: POINTER TO RECORD END) by its name. Therefore it is not possible to register mappers for an anonym type. The automatic mapper is always used. This restriction should not be a big problem, because objects of an anonym type are most times used only as local variables and surely not as persistent data.
The type descriptor contains no information about the record components, with the exception of pointers. Therefore an automatic mapper writes and reads the memory block as it is, with no different treatment of e.g. integer, long integer or procedure variables. This means that it is not possible to take the persistent heap from a computer with little endian format to a computer with big endian format or vice versa. As well it is not possible to use automatic mappers for a record containing a procedure variable. A solution for these problems would be to extend the reference information, generated by the Oberon compiler (see [Tem94, p.93f]).
3 Implementation
This section describes some interesting parts of the implementation of Oberon-D. For details see the source code. Modules and operations of the Oberon system can be studied in [Rei91].
3.1 Overview
Figure ??? shows the module hierarchy of Oberon-D.
The programming interfaces of the modules OberonD and Persistent were shown in the sections
and
. The other modules are more for internal use. Their programming interfaces will be shown in
A first decision had been made to define how to identify a persistent object, what means how to distinguish two different persistent objects. This is managed by the object identifier (OID), a unique key for each persistent object.
To make an object persistent it is necessary to map the object to an external representation. This process is declared in this and the following section. There are three main points to decide in this process: When should the object mapped, where should it survive, and how works this process?
When?
This point was easy to decide, because of the implementation of the Oberon memory management, there is only one possible time to map these objects. The Oberon system uses the Mark & Sweep [.???] garbace collector algorithm, what means that at the moment after the Mark phase and before the Sweep phase is the right time to map an unmarked persistent object. Each moment before would mean that the object could be changed after mapping, because there could be existing references to it. After Sweep, mapping the object would be impossible, because the object does not longer exist.
Where?
All persistent objects are stored in one persistent heap, defined by the variable PersKernel.heap at the position x, given through the OID of the object. In future versions of Oberon-D it should be thought over the existence of different heaps (see also section
Mapping an object between Mark and Sweep is accompanied with different problems (one is not allowed to allocate memory or to use methods), therefore a simple technique was used for this finalization [Tem94, p.83ff]. The Oberon module Kernel (responsible for the Oberon memory management) offers a procedure Kernel.RegisterObject (obj, fin) where fin is a procedure variable of type Finalizer = PROCEDURE (obj: SYSTEM.PTR) and obj is an object which should be finalized before being reclaimed by the garbage collector. The module Kernel checks between Mark and Sweep, whether there is an registered object which sould be reclaimed and marks it in a special way. After the sweep phase, Kernel calls the registered finalization procedures for all special marked objects. The implementation of this finalization strategy is shown in [Tem94, p.87f] and was a little change in the Oberon system (see also section
The job of Oberon-D is now to register the finalization procedure Persistent.Fin for all persistent objects, which maps an object to an external presentation. This procedure is explained in section
Another interesting point is the loading of a persistent object. This has to be done at the time of an access to a property of the persistent object. Before such an access the loading is not necessary and any transient reference to such an object contains the negative OID of the object. Before such accesses the Oberon compiler generates NIL-Traps [????], which are causes if the object is equal NIL. A little change to the compiler (see section
) causes the trap also if the object contains a value lower than zero. Installing an own trap handler (Persistent.Trap) allows Oberon to load the object at this time. The implementation of the trap handler is explained in section
3.2 Finalization
3.2.1 Overview
The finalization is managed through the procedure Persistent.Fin, called by the Oberon module Kernel. This procedure has to do following things:
Determining the object's OID. For this purpose a relation of memory addresses to OIDs must be available. This relation is stored in a BTree (see Appendix
After that a Files.Rider [Rei91] has to be set on PersKernel.heap at the position x, defined by the objects OID.
Writing a start sign to the heap. The start signs determines wheter the object is an array (PersKernel.array) or a record (PersKernel.record).
Writing the type of the object to the heap. In most cases only the type number is written to the external memory (for details see below). This type information is necessary for loading the object back.
Writing the object information to the heap. This means to call the (automatic or registered) mappers for each type extension level. If the object is an array, also the array information (number of dimensions, number of elems for each dimension) has to be written to the heap. Three different cases are possible by the storage of arrays, they are mentioned below.
Writing the object information to the external memory is managed by procedures of Persistent (Write, WriteInt ...). In most cases these procedures are calling the corresponding Files procedures (e.g. Files.Write, Files.WriteInt ...), but there are two different special cases:
WriteObj (o1) writes only the OID of the object o1 to the persistent heap. If o1 is not a persistent object it will be marked as a persistent object (see section
WriteProc (proc) writes the name of the procedure proc and the module defining the procedure to the persistent heap.
3.2.2 Automatic Generation of Object Mappers
The automatic storage can be done by the procedure PersSys.Automatic. As explained above the module PersSys is a system-dependent module, what means that the automatic generation of mappers depends on the underlying system. This, because the type information is necessary, and this information is not exactly equal on all Oberon implementations.
The idea of an automatic mapper is simple. Figure ??? shows the type information offered by the Oberon system (in the shown case: PowerMac-Oberon).
Figure ???. Type descriptor
PersSys.Automatic for a type t has now to take the memory area of the fields introduced by type t and writes it to the heap byte by byte, with the exception of pointers. These fields can be extracted with the using of the type information (ptroff x) and are stored with Persistent.WriteObj.
3.2.3 Storage of objects of an anonym type
Objects of an anonym type are stored with the help of automatic generated mappers. Because of the impossibility to identify an anonym type by number or name, the type information has to be stored in the heap. So the size of the object on persistent and transient heap, an eventually existing basetype, the module defining the type, and the pointer offsets have to be stored, too.
3.2.4 Storage of Arrays
The programming language Oberon offers also the possibility to allocate open arrays on the heap. An open array is a pointer to an array of type (e.g. POINTER TO ARRAY OF CHAR). Depending on the type three different kinds of open arrays can be distinguished (Array of Record, Array of Pointer, Array of simple type). The necessary information about these arrays is available through the array descriptors offered by the Oberon system (see fig. ???). The construction of these descriptors depends on the undlerlying system (in the shown case PowerMac-Oberon is used).
Figure ???. Array descriptors
For the reconstructing of the stored arrays in the loading phase Oberon-D needs the following information:
number of dimensions
length of each dimension
type of the array elements
After storing this information, each element of the array has to be mapped, as described above, like an ordinary persistent object.
3.3 Loading of persistent objects
As mentioned above the trap handler Persistent.Trap is responsible for the loading of persistent objects. Because of a little change in the Oberon compiler a NIL-trap [????] causes by an access to a not-loaded persistent object. The job of Persistent.Trap is now to determine the register which contents causes the NIL-trap. If the contents of the register is equal zero than the standard trap handler must be called, otherwise the absolute value of the register contents is equivalent to the persistent object's OID. In this case the trap handler has to load the persistent object in the transient heap. To reach this aim the operation Persistent.Load is responsible. This procedure has to do the following things:
Determining that the object is not already loaded: For this purpose a relation of OIDs to memory addresses must be available. This relation is stored in a BTree (see Appendix
). If the object is already loaded the contents of the register is set to the memory address of the object.
If the object is not already loaded, a Files.Rider [Rei91] has to be set on PersKernel.heap at the position x, defined by the object's OID.
Reading of the start sign and the type information from the heap (equivalent to writing, see section
Allocating of an object of the read type.
Reading the object information from the heap (equivalent to writing, see section
The loaded object has to been marked as a persistent object. This means it must registered in the two relations (memory addresses to OIDs and OIDs to memory addresses) and in the Kernel finalization queue.
Setting the contents of the register to the memory address of the object.
As mentioned above, only the register contents is set to the memory address. This means, that a next access to this persistent object, causes another trap. To prevent this, three optimiziation strategies are implemented:
Backwards scanning of the machine instructions, to get the load instruction where the patched register is loaded from memory. Thereby is is possible to get the memory address of the pointer and set the memory pointer to the correct value, preventing further traps. Unfortunately there are some algorithms (e.g iterating a list) where the loading instruction is not easily found. Also this optimization is highly system-dependent.
Any persistent object may have references to other objects. If now an object is loaded into memory it is checked wether one or several of its references is an OID of an already loaded object. In that case the OID is replaced with the actual pointer value. Thereby further traps, when dereferencing this pointer, are prevented.
The last optimiziation strategy keeps a list (cache) of the lastest n loaded objects. When a trap appears, the requested persistent object is loaded and an iteration over this list shows whether these objects have reference the newly loaded object. If this is the case, the references are set to the correct memory address. The two last optimiziation strategies are especially suited for iterations over lists and other similiar algorithms.
These three optimiziation strategies prevent a considerable number of traps. Most test cases showed almost optimal behaviour.
Another problem is the correct setting of global references, to prevent loaded persistent data to be freed by the garbage collector. To prevent this, we introduce a new phase in the garbage collector: the prepare phase (see also section
). The gargbage collector offers a list of procedures Kernel.prepQ which are handled before the mark phase is started. Any user can register his own procedures. Oberon-D registers the procedure PersSys.SetGlobPersVars, which scans all global pointers to find references containing the negative OID of an already loaded object. These references are set to the correct memory addresses.
3.4 Persistent Memory Management
The persistent memory management consists of the root mechanism, the list of persistent types, the necessary processes to make an object persistent, and the persistent garbage collector.
3.4.1 Persistent Roots
Oberon-D makes everything persistent what is reachabe through a persistent root. For this purpose a list of the persistent roots is maintained by Oberon-D. This is done by the module PersKernel with the help of a simple list, sorted by the root-key. The list can be iterated through the procedure IterateRoots. When initializing the database, this list is loaded from an simple Oberon file, which is updated on every change to the list.
3.4.2 Persistent Types
Oberon-D needs special information about the type of a persistent object. This information is stored in the module PersTypes, which maintaines a list of persistent types. The type information contains the type name, the inclosing module, a unique number (for a shorter type identification), the registered read and write mappers (if existing), the size, and the pointer offsets (on the disk). If mappers are registered, the size and pointer offsets can differ between the transient and persistent representation. The pointer offset information is needed for the persistent garbage collection (see section
If an object of a type, not contained in the list of persistent types, is made persistent, the type is added to this list. Default values, e.g. automatic mappers, will be taken as properties. Adding to the list with non standard values, is possible through the function Persistent.RegisterType, which calls the function PersTypes.Register (see section ???).
3.4.3 Making a transient object persistent
If a transient object is made persistent (through a call of Persistent.WriteObj or Persistent.SetRoot) several things are done:
Determining the persistent type pt (see section
) of the object.
Allocating space on the persistent heap.
Register the object in the finaliziation queue (see sections
and
Register the object's OID and its memory address in the two relations (memory addresse->OID) and (OID->memory addresse).
3.4.4 Persistent Garbage Collection
Oberon-D uses the garbage collection algorithm Stop & Copy [????] to delete obsolete persistent data. This algorithm (implemented in PersKernel.GC) uses two heaps (files) and copies all accessible object from the heap fromHeap to the heap toHeap. The algorithm consists of three steps:
Copy all root objects (by iterating the root list with IterateRoots) (see fig. ????).
Move Scan-Pointer through objects: Copying of all accessible object from the fromHeap to the toHeap until scan = free (see fig. ???).
Problem: pointers from toHeap to already copied objects in fromHeap (e.g. cyclic structures).
after copying: mark old object in fromHeap by overwriting it with pointer of copied object.
pointer from the fromHeap to the toHeap: bend to copied object (see fig. ???.).
Swap fromHeap and toHeap.
This algorithm can be optimized by caching fromHeap and toHeap in the transient memory. Nearly the complete memory can be used for this purpose.
3.5 Programming Interfaces
3.5.1 PersKernel
Module PersKernel is responsible for the persistent memory management and the list of persistent roots. It allocates memory on the persistent heap, sweeps garbage from the persistent heap, and allows to set and remove persistent roots.
PROCEDURE RegisterRoot (str: ARRAY OF CHAR; oid: LONGINT);
PROCEDURE RemoveRoot (str: ARRAY OF CHAR);
END PersKernel.
State
heap is the file containing the persistent heap.
heapSize is the current size of the persistent heap.
number is the current number of persistent objects.
res indicates the success of an operation.
Operations
GC starts the persistent Garbage Collector (see also OberonD.GC).
NewSys (oid, s) allocates persistent memory for an object of size s. The object's OID is retrieved in oid.
OID (str) returns the object identifier of the persistent root object given by the key str.
RegisterRoot (str, oid) registers the persistent object with object identifier oid as a persistent root with key str. The list of roots is automatically stored after a new root has been registered.
RemoveRoot (str) removes the persistent root with key str. Afterwards, the object referenced by the root str still exists, but only as a transient and not anymore as a persistent object. The list of roots is automatically stored after a root has been removed.
IterateRoots (proc) iterates the list of persistent roots and calls the procedure variable proc for each found root.
3.5.2 PersTypes
Module PersTypes manages the persistent types. This module corresponds to the module Types, which manages the transient types. PersTypes is necessary for the storing of the unique type number, the read and write mappers, and the pointer offsets on the persistent heap.
DEFINITION PersTypes;
IMPORT SYSTEM, Types, Modules;
CONST
ok = 0; mapperNotRegistered = 4; (* error codes *)
TYPE
Mapper = PROCEDURE (o: SYSTEM.PTR);
PtrArr = POINTER TO ARRAY OF LONGINT;
Type = POINTER TO TypeDesc;
TypeDesc = RECORD
nr-: INTEGER; (* unique type number *)
name-, module-: ARRAY 32 OF CHAR; (* name of type, defining module *)
auto-: BOOLEAN; (* automatic or registered mappers *)
rmap-, wmap-: Mapper; (* mappers *)
size-: LONGINT; (* size of an object on the persistent heap *)
nofptrs-: INTEGER; (* number of pointers on the persistent heap *)
ptrArr-: PtrArr; (* locations of the pointers *)
END ;
res: INTEGER;
PROCEDURE Register (t: Types.Type; read, write: Mapper; size: LONGINT; VAR ptrArr: ARRAY OF LONGINT);
PROCEDURE This (mod, name: ARRAY OF CHAR): Type;
PROCEDURE ThisByNr (nr: INTEGER): Type;
END PersTypes.
Operations
ThisByNr (nr) returns the persistent type with the unique number nr.
This (mod, name) returns the persistent type corresponding to the transient type name defined in module mod.
Register (t, read, write, s, ptrArr) registers a persistent type identified by t with the mappers read and write. An object of this type will need s bytes on the persistent heap and its pointer offsets are given by ptrArr. PersTypes generates a unique number as the access key for this type. This procedure is called by Persistent.RegisterType.
3.5.3 PersSys
PersSys contains all system dependent operations of Oberon-D. Porting Oberon-D from one system to another (e.g. from PowerMac-Oberon to Windows-Oberon) means to port this module. The following operations are only for internal use.
Browser.ShowDef PersSys ????
Operations for scanning the reference information
FindProc (adr, mod, proc) returns the procedure information (mod, proc) of the procedure starting at the address adr.
StartPC (mod, proc) : LONGINT returns the starting address of the procedure defined by (mod, proc).
Trap Handling
NilTrap (ctx) returns TRUE, if the trap described by the context ctx was produced by a nil-Trap [???].
UnmappedMemoryTrap (ctx) returns TRUE, if the trap desribed by ctx was produced by an unmapped memory Trap [????].
Type Information
AnonymType (m, bt, ptr, nofptrs, s) returns an anonymous type, defined in module m, derived from type bt, with nofptrs pointers, described by ptr, and with object size s.
GetPtr (t, ptrArr, nofptrs) retrieves the number of pointers nofptrs from type t, and their offsets in ptrArr.
NewObj (o, t) retrieves a new object o of type t. The contrast to Types.NewObj, no type test on the static type of o is done.
Size (t) returns the necessary size (on the transient heap) for an object of type t.
Array Information
CheckIfArray (o, cond, t) checks if o is a dynamic array and retrieves the result in cond. Furthermore the type of o, if it is not an array, or , if it is an array, the type of its elements, is retrieved in t.
GetArrInf (o, first, last, nofelem, s) retrieves following array information about the array o: The number of array elements nofelem, the memory borders first and last, and the necessary size on the transient heap s.
NewArr (t, dim, nofdim, new, first, last) allocates a new array at address new. Its dimensions are described by nofdim and dim. The array elements are of type t. The memory borders first and last are retrieved.
ReadSimpleArr (r, o) reads a simple dynamic array (e.g. ARRAY TO POINTER OF CHAR) o, meaning array information and contents, from rider r.
StoreArrInf (r, o, first, last, nofelem) stores the array information of array o on the rider r. The number of array elements nofelem, and the memory borders first and last are retrieved.
Optimization
SetGlobPersVars scans all global pointers to find references containing an OID of an already loaded object and sets them to the correct memory addresses.
Miscellaneous
SetStaticBase (proc, m) sets the static base of the procedure variable proc to the right value, determined by the properties of module m. This is necessary with eight bytes sized procedure variables, e.g. in PowerMac-Oberon.
MapAutomatic (r, o, t, read, mark) maps the object o, which is of type t, to the rider r. read determines whether the object should be read or written. mark determines whether the system is in the mark-phase or not. If read is FALSE and mark TRUE, the object is not really written, but all referenced objects are marked as persistent.
3.6 Necessary changes in the Oberon System
3.6.1 Compiler
The loading of persistent objects into the transient heap is managed by the trap handler (see
). Therefore, each access to a property of a not yet loaded persistent object must cause a trap. Because of the existing NIL-Check (twi in the compiler implementation for the PowerMac) before such accesses, we decided to use this trap. We changed the NIL-Trap such that it causes a trap when a pointer contains a value equal or lower zero. Lower zero means, that the object is persistent and contains a negative OID (see
). This is just a small change to one line of the Oberon compiler [Cre90].
3.6.2 Kernel
Object finalization has been made safe. Any Oberon application may now register objects for finalization, together with a finalization procedure. When the garbage collector reclaims memory, it first calls the registered finalization procedure, which may perform cleanup operations.
The module Kernel exports a new procedure RegisterObject (obj: SYSTEM.PTR; fin: Finalizer) and a new type Finalizer = PROCEDURE (obj: SYSTEM.PTR).
The object finalization implementation is based on [Tem94, p. 83ff].
Procedure queues have been added. A procedure queue is a list of procedures which are handled at a well defined moments. There are four different queues, handled at different times: prepQ (handled at the beginning of the garbage collector), gcQ (between mark and sweep phase), closeQ (after the garbage collector) and quitQ (before leaving the Oberon system).
The transient garbage collector (Kernel.GC) had to be adapted, as global variables and heap addresses can now contain values lower than zero.
4. Conclusions
Oberon-D is an ongoing project. Persistence was but the first step. There are other goals, like platform-independence, simplicity and extensibility. With the exception of some mentioned non-portable parts (see sections
and
), these goals have been reached. The next steps will be the inclusion of other database functionalities in the Oberon system:
Schema Evolution: Many object oriented database systems allow the user to modify type definitions. However, they vary considerably in the amount of assistance they offer in handling such modifications. An example: When adding a new attribute to an object type, is it necessary to explicitly "fix" all existing objects of the changed type to include the new attribute? Is it possible to add a new supertype when instances of a type exist? [Cat94, p.116ff].
Recovery: Recovery mechanisms allow a consistent state of the database systems to be recovered after a system crash.
Query Languages: Query languages are important functionalities of database systems. The user can retrieve data by specifying the conditions the data must meet. In relational database systems, query languages are the only way to access data, wheras object-oriented database systems, in general, have two ways. The first, in Oberon-D already implemented, is navigational and exploits object identifiers and aggregation hierarchies. Given an OID, the system accesses the object directly and it navigates through the objects referred to by its attributes. The second is access through a query language.
Concurrency control limits simultaneous reads and updates by different users, to give all users a consistent view of the data [Cat94, p.69ff]. Although Oberon-D is an one-user database system, there may be different tasks working simultaneously on the persistent heap.
Additionally the inclusion of persistence in the Oberon system, may be improved in the future. Some ongoing directions are:
Optimiziation of the persistent garbage collection.
Storage of extended reference information to avoid some restrictions mentioned in section
Usage of more than one heap.
Porting Oberon-D to additional platfoms besides Windows and PowerMac (e.g. Sparc, Linux, ...).
Acknowledgments
I wish to thank Prof. Hanspeter M
ssenb
ck and Prof. Gerti Kappel for their support of this project. My thank goes also to Markus Hof, Christian Mayrhofer, Christoph Steindl and Josef Templ for many stimulating discussions about Oberon-D. Markus Hof - sitting on the other side of my desk - was also responsible for porting Oberon to the PowerMac, and therefore could give me necessary information to implement the module PersSys. Josef Templ's PhD thesis "Metaprogramming in Oberon" was a rich source of inspiration for Oberon-D.
version, nofObjects, size, realSize, baseTypenr, ptroff, dim, oid are identifiers of Oberon type LONGINT.
byte is an identifier of Oberon type SYSTEM.BYTE.
modName and mapper are identifiers of type ARRAY 32 OF CHAR.
recSign and arrSign are exported constants of module PersKernel (PersKernel.record and PersKernel.array).
Appendix
???: ISFiles
Oberon-D uses several B-Trees [BaMc72], e.g. for the relation (memory address->OID). These trees are implemented in the module ISFiles.
This module provides a set of routines for 'index sequential files'. An index sequential file consists of (key, data) pairs. With help of the key, one gets the corresponding data. The length of the key and data is arbitrary and is defined when a new file is created.
PROCEDURE Create (name: ARRAY OF CHAR; keySize, dataSize: INTEGER): File;
PROCEDURE Delete (f: File; VAR key: ARRAY OF SYSTEM.BYTE);
PROCEDURE Open (name: ARRAY OF CHAR; mode: INTEGER): File;
PROCEDURE Read (f: File; VAR key, data: ARRAY OF SYSTEM.BYTE);
PROCEDURE ReadNext (f: File; VAR key, data: ARRAY OF SYSTEM.BYTE);
PROCEDURE Restart (f: File);
PROCEDURE Write (f: File; VAR key, data: ByteArrDesc);
END ISFiles.
Operations
Create (name, keySize, dataSize) creates a new index sequential file with the name name. The file is opened for read and write operations.
Open (name, mode) opens an existing index sequential file name. The parameter mode specifies the access mode. There are two different modes. readOnly, if only read operations, or readWrite, if read and write operations are intended.
Close (f) closes the file f opened previously with Create or Open.
Read (f, key, data) reads the data associated with key and retrieves it in data. If an error occured, the error code is stored in f.status. Otherwise, f.status is ok. Possible errors are errAccessMode, if the file was previously closed or errIllegalKey, if there exists no data with the given key.
ReadNext (f, key, data) reads the data and key from the file f with the smallest key bigger than the key of the most recently read data. If an error occured, the error code is stored in f.status. Otherwise, f.status is ok. Possible errors are errAccessMode, if the file was previously closed or errRecNotFound, if no further data with a bigger key exists.
Restart (f) resets the file f back to its start. A following ReadNext operation returns the first (key, data) pair in the file (data with smallest key).
Write (f, key, data) overwrites the data - associated with the key key - with data. If no pair with this key exists, a new one is generated. If an error occures, the error code is stored in f.status. Otherwise, f.status is ok. Possible errors are errAccessMode, if the file is not open for write access or errDiskFull, if the block couldn't be allocated.
Delete (f, key) deletes the pair with the key key from the file f. If an error occures, the error code is stored in f.status. Otherwise f.status is ok. Possible errors are errAccessMode, if file is not open for write access or errIllegalKey, if no block with this key exists.
Errors
All operations may not succeed with there task. If a problem occurs, an error is returned. With Open and Create an error is indicated by a return value of NIL. All other functions use the File property status to indicate their success or failure.
Appendix
???: Error-Codes
0: Ok, no error.
1: This root already exists.
2: There is no root with a corresponding name.
3: The database must be offline (before starting persistent garbage collection).
4: The mappers were not registered.
Appendix ??: How to get Oberon-D
Oberon-D is part of the Oberon version for the PowerMac. This distribution can be obtained via anonymous internet file transfer ftp (at no charge).
FTP Hostname: Oberon.ssw.uni-linz.ac.at
FTP Directory: /pub/Oberon/PowerMac
Appendix ???: References
[AgGe89] R.Agrawal and N.H.Gehani,
"Ode (Object Database and Environment):
The Language and the Data Model"
Proc. ACM_SIGMOD 1989 Int'l Conf. Management of Data, Portland,
Oregon, May - June 1989, pp. 36-45
[AtBu87] M.P.Atkinson, O.P.Buneman,
"Types and Persistence in Database Programming Languages"
ACM Computing Surveys, Vol. 19, No.2, June 1987, pp. 105-190
[BaMc72] R.Bayer, E.M.McCreight
"Organization and Maintenance of Large Ordered Indexes"
Acta Informatica, 1, No. 3 (1972), pp. 173-189
[Cat94] R.G.G.Cattel, "Object Data Management"
Addision-Wesley, 1994
[Cre90] R.Crelier, "OP2: A Portable Oberon Compiler"
(.txt)